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Abstract 

Tradespace  exploration  supports  the  Systems  Engineering  Technical  Management  Process  of  Decision  Analysis  by  identifying 
compromises,  revealing  opportunities,  and  communicating  the  impacts  of  decisions  across  a  system’s  development  lifecycle. 
Critical  program  decisions  are  made  based  on  the  outcomes  of  trades;  trades  being  performed  with  multiple  types  and  quantities 
of  data  coming  out  of  tools  and  methods  employing  qualitative  and  quantitative  analyses.  Tradespace  exploration  for  Engineered 
Resilient  Systems  (ERS)  is  envisioned  to  coalesce  pertinent  infonnation  tuned  to  specific  decision  makers,  at  the  appropriate 
time,  presenting  a  holistic  view  of  decision  impacts  on  required  system  capabilities.  This  study  provides  an  ERS  view  of 
tradespace  exploration,  which  reveals  that  having  a  valid  set  of  attributes,  and  an  understanding  of  how  a  cross-section  of  tools 
can  satisfy  them,  is  insufficient  -  what  is  needed  is  a  deeper  understanding  of  how  these  tools  are  used  and,  more  importantly, 
how  they  can  be  used  when  performing  tradespace  exploration  in  support  of  the  Decision  Analysis  Process.  Gaining  this 
understanding  will  enable  users  to  better  assess  if  they  possess  the  appropriate  tradespace  exploration  tools.  A  holistic  view  of  8 1 
candidate  tradespace  exploration  tools  is  provided.  This  study  seeks  to  address  a  fundamental  aspect  of  tradespace  exploration  by 
assembling  a  “best  common  practice”  process  for  their  requirements,  identifying  a  set  of  attributes  that  defines  an  ideal 
tradespace  exploration  tool,  and  surveying  existing  tools  that  satisfy  these  attributes.  In  this  way,  a  set  of  tools  can  be  selected  to 
enable  the  ERS  tradespace  vision  on  a  particular  project.  A  paradigm  shift  towards  common  tradespace  methods,  tools,  cost 
models,  and  steps  is  emphasized. 
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1.  Background 

As  the  Department  of  Defense  (DoD)  moves  towards  addressing  geopolitical  environments  marked  by  rapidly 
changing  threats,  tactics,  mission  scenarios,  technologies,  and  available  funding,  advanced  methods,  processes,  and 
tools  are  needed  to  effectively  engineer  resilient  system  solutions.  The  resulting  resilient  systems  must  be  adaptable 
to  a  wider  range  of  mission  contexts,  across  multiple  alternative  futures.  To  support  resilient  system  design  -  and  a 
correspondingly  resilient  design  process  -  tradespace  exploration  (TSE)  provides  decision  makers  an  understanding 
of  capabilities,  gaps,  and  potential  compromises  facilitating  the  achievement  of  system  metric  objectives.  The 
decisions  being  made  throughout  a  system’s  development  lifecycle  are  continuously  redefining  its  capabilities, 
performance,  cost,  manufacturability,  delivery  schedule,  and  sustainability.  To  be  effective,  decision  makers  must 
have  deep  knowledge  of  the  component  elements  of  a  system,  including  how  these  elements  interact  internally  to  the 
system  and  externally  with  the  operational  environment.  TSE  provides  decision  makers  an  understanding  of 
candidate  system  component  choices  and  the  implications  on  multiple  missions  across  joint  Warfighting 
environments'. 

Throughout  a  system’s  development  lifecycle,  and  across  its  hierarchy,  trades  are  being  performed  with  multiple 
types  and  quantities  of  data  coming  out  of  varying  levels  of  analysis  fidelity.  From  a  system’s  functional 
perspective,  capabilities  (e.g.,  lighter  weight,  higher  performing)  map  to  system  measures  of  performance,  which  in 
turn  map  to  measures  of  effectiveness,  which  ultimately  affect  operational  figures  of  merit  that  determine  how  well  a 
system,  or  a  portfolio  of  systems,  meets  a  required  capability.  Concurrently,  there  is  a  product  hierarchy  within 
which  various  technologies  can  be  applied,  from  the  component  level,  and  up  to  subsystem,  system,  and  finally  a 
portfolio  of  heterogeneous  systems  working  concurrently  to  fulfill  a  role.  Additionally,  there  are  process 
considerations  such  as  industrial  base,  training,  policy,  and  procedures,  which  are  often  omitted  from  the  decision 
analysis  process  for  various  reasons.  Although  the  decisions  being  made  throughout  these  functional,  product,  and 
process  perspectives  differ,  decision  makers  at  the  highest  levels  will  execute  based  on  the  relationship  between  the 
benefit  achieved  by,  and  the  lifecycle  cost  (LCC)  associated  with,  a  particular  asset  mix.  Therefore,  if  the  tradeoff 
between  benefit  and  cost  can  be  generated,  explored,  and  then  presented  in  the  context  of  the  holistic  system,  a  more 
informed  decision  can  be  made  across  the  system  perspectives. 

To  aid  the  decision  maker  in  performing  effective  TSE  under  these  complex  circumstances,  the  ERS  effort 
involves  providing  the  necessary  engineering  concepts,  methods,  processes,  and  tools^.  The  goals  of  ERS  are  to 
provide  to  engineering,  warfighting,  and  acquisition  decision  makers  the  needed  capability  to  manage  TSE  activities 
with  full  and  consistent  information  throughout  the  life  of  the  systems  by^: 

•  Producing  more  complete  and  robust  requirements  pre-Milestone  A 

•  Making  the  engineering  design  process  much  more  efficient  and  effective 

•  Considering  the  manufacturability  of  a  proposed  design  explicitly 

•  Establishing  baseline  resiliency  of  current  capabilities 

The  TSE  methods,  processes,  and  tools  should  enable  deeper  consideration  of  system  design  alternatives  while 
keeping  the  space  as  open  as  possible  to  address  resiliency  and  robustness  to  changing  conditions  and  constraints. 

The  tradespace  frontier  of  Fig.  D  depicts  that  information  coming  out  of  knowledge  databases  must  support 
decision  making  across  the  lifecycle  by  communicating  to  multiple  perspectives  and  across  the  system  hierarchy, 
while  taking  into  account  fiscal  and  environmental  constraints.  What  this  means  is  that  TSE  tools  and  processes  are 
needed  at  different  levels  of  the  product  hierarchy,  commensurate  with  the  decisions  being  made,  the  information 
available  to  make  decisions,  and  the  person  or  organization  making  decisions. 
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Fig.  1.  ERS  tradespace  frontier 


2.  Assumptions 

•  TSE  is  performed  throughout  a  system’s  lifecycle,  in  support  of  decision  making,  in  a  step-by-step  process. 

•  There  exist  organization-specific  best  practice  methods,  processes,  and  tools  for  performing  TSE. 

•  TSE  is  often  performed  to  justify  selection  of  a  reduced  set  of  options  from  a  broader  set  of  alternatives,  rather 
than  for  understanding  how  compromises  in  system  design  can  increase  value  delivered  to  stakeholders. 

•  TSE  is  largely  performed  ad  hoc,  and  with  the  tools  on-hand,  without  investigation  into  how  other  tools  can  be 
brought  in  to  fill  gaps  in  a  TSE  process. 

•  TSE  is  performed  in  order  to  “make  better  decisions”,  but  without  criteria  for  measuring  the  effectiveness, 
efficiency,  correctness,  or  consistency  of  decisions  across  the  lifecycle. 

3.  Approach 

To  reveal  whether  tradespace  exploration  best  practices  are  on  a  trajectory  to  meet  the  needs  of  ERS,  sources  of 
TSE  methods,  processes,  and  tools  were  identified,  common  approaches  in  TSE  were  documented,  a  set  of  TSE  tool 
“requirements”  was  developed,  the  most  commonly  used  TSE  tools,  as  well  as  other  tools  that  could  be 
implemented  in  a  formal  TSE  process,  were  identified,  and  an  evaluation  was  performed. 

When  making  a  decision,  in  this  case  regarding  the  tools  to  use  in  exploring  a  tradespace,  a  common  approach  is 
to  start  with  alternatives  and  determine  which  alternative  best  meets  the  decision  maker’s  intent.  A  preferred 
approach  is  one  based  on  value-focused  thinking,  where  value-producing  objectives  are  developed,  alternatives  are 
identified,  and  decisions  are  made  based  on  how  much  value  the  alternatives  provide  across  the  objectives''.  Applied 
formally,  a  value-focused  thinking  approach  involves  interviews  with  decision  makers  to  help  translate  the 
perceived  increase  in  an  evaluation  metric  into  a  value  curve.  In  this  study,  a  modified  value-focused  thinking 
approach  was  started;  a  recommendation  will  be  made  for  expanding  the  current  approach  to  include  generating 
value  curves  and  assigning  swing  weights  to  value  measures  for  application  to  an  additive  value  model. 

To  initiate  a  value-focused  thinking  approach,  a  fundamental  objective  was  developed  assuming  that  TSE  is  a 
process  that  uses  tools  to  arrive  at  decisions:  ‘Use  a  tradespace  tool,  or  set  of  tools,  to  make  actionable  decisions 
across  the  system  hierarchy  and  throughout  the  system  lifecycle.’  In  order  to  evaluate  a  tool’s  performance  within  a 
TSE  process,  several  pieces  of  information  were  needed:  TSE  tools,  TSE  process  steps,  attributes  that  define  TSE 
tools,  and  attributes  that  define  what  users  are  doing  with  their  TSE  tools. 
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4.  Tradespace  Analysis  Tool  Requirements  Identifieation 

An  audit  was  performed  to  identify  the  existing  TSE  tools  used  by  the  ERS  Demonstration  Projects  (DP)',  as  well 
as  elsewhere  in  the  Services  and  Industry,  throughout  a  system’s  lifecycle.  A  set  of  attributes  that  define  various 
TSE  tools  was  developed,  and  the  tools  were  mapped  to  the  attributes  to  aid  in  identifying  existing  tools  that  are 
closely  aligned  with  needed  capabilities. 

4.1.  TSE  Tools 

The  sources  shown  in  Table  1  were  used  to  identify  TSE  tools.  After  completing  the  audit  of  tools,  it  was 
apparent  that  what  most  refer  to  as  a  decision  analysis  or  data  mining  tool  could  be  misconstrued  as  a  satisfactory 
ERS  TSE  tool.  To  better  understand  how  a  decision  analysis  tool  is  viewed,  a  definition  of  Decision  Analysis 
developed  by  Buede*,  and  referred  to  in  the  2010  Operations  Research  /  Management  Sciences  (OR/MS)  Today 
survey  summary*,  is  provided:  “Decision  analysis  is  the  discipline  of  evaluating  complex  alternatives  in  the  light  of 
uncertainty,  value  preferences  and  risk  preference.”  Although  the  process  of  Decision  Analysis,  as  described  in  the 
Defense  Acquisition  University’s  (DAU)  Defense  Acquisition  Guidebook  (DAG)'^,  is  not  TSE  in  and  of  itself,  it  is 
supported  by  TSE.  In  the  case  of  data  mining  tools,  the  corresponding  Survey"  does  not  provide  a  formal  definition, 
but  similarities  to  TSE  activities  are  apparent  with  terms  such  as  regression,  models,  and  data  visualization. 

Table  1.  TSE  tool  sources 
TSE  tool  source 

Operations  Research  /  Management  Sciences  (OR/MS) 

Today  Decision  Analysis  Software  (DAS)  Survey^’^ 

Rexer  Analytics  Data  Miner  Survey^ 

ERS  DP'^’*"’" 

ERS  Priority  Steering  Council  (PSC)  Tools  Assessment'^ 

ARE  internal  organization  research  and  experience 


The  ERS  PSC  assessment'"  was  filtered  for  tools  in  the  “Data  Driven  Tradespace  Exploration  and  Analysis” 
section.  In  addition  to  the  source  materials  in  Table  1,  the  ERS  DP  leads  provided  overview  presentations  and  were 
informally  interviewed  via  telephone.  These  sources  all  revealed  that  there  are  many  tools  in  use  across  academia, 
government,  and  industry  for  performing  TSE.  In  all,  81  tools  were  identified  (the  authors  recognize  that  more  tools 
exist  than  are  captured  here). 

4.2.  TSE  Tool  Reduction 

After  identifying  a  large  number  of  tools  for  initial  consideration,  the  next  step  was  to  reduce  the  set  to  a 
manageable  size,  knowing  that  the  subsequent  tasks  would  involve  identifying  attributes  and  then  mapping  tools  to 
those  attributes  in  an  attribute-by-tool  sized  matrix.  A  set  of  keep-or-reject  criteria  was  developed.  The  first  criterion 
was  a  gate:  if  the  tool  was  in  use  by  any  of  the  ERS  DP  teams  then  it  was  kept  for  consideration  in  the  present  study, 
and  if  not  then  it  entered  into  a  set  of  filter  criteria.  The  first  filter  criterion  was  whether  the  tool  or  developer  was 
still  available;  in  several  cases  an  Internet  search  for  the  tool  or  developer  returned  limited  results  or  no  web  page. 
The  next  filter  criterion  was  whether  the  tool  was  included  as  part  of  a  larger  suite  of  tools;  in  several  cases 
individual  tools  were  rolled  into  a  larger  suite  by  the  developer,  in  which  case  the  suite  was  assessed  for  evaluation 
as  opposed  to  the  individual  tool.  The  final  filter  criterion  was  if  the  tool  was  specific  to  a  domain  outside  of  system 
design  and  analysis  (e.g.,  medical  or  service  industry);  tools  with  a  niche  function  were  omitted  in  order  to  focus  on 
tools  encompassing  a  broad  TSE  process.  This  initial  filtering  reduced  the  tool  set  by  38. 

The  remaining  43  tools  were  reassessed  to  determine  if  they  should  be  evaluated  in  the  current  effort  or  deferred 
to  a  follow-on  effort.  The  first  filter  criterion  at  this  level  was  if  the  tool’s  development  or  use  was  supported  by  any 
known  DoD  TSE  effort,  or  if  it  is  receiving  DoD  funding  for  continued  development  and  application;  if  so,  then  it 
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was  kept  for  current  consideration.  The  next  filter  criterion  was  if  the  tool  received  high  ranking  and/or  positive 
feedback  from  other  tool  surveys;  several  tools  showed  consistent  or  rising  popularity,  which  the  authors  assessed  as 
indicative  of  high  user  satisfaction.  The  final  filter  criterion  was  if  the  authors  had  any  familiarity  with  the  tool 
whatsoever;  if  none,  then  the  tool  was  deferred  for  assessment  since  the  only  other  information  available  was 
developer  marketing  and  advertising  literature,  and  the  authors  felt  it  inappropriate  to  rely  on  secondary  knowledge 
to  complete  the  initial  assessment.  Using  these  criteria,  the  subset  of  43  tools  was  reduced  to  13.  Regardless  of 
which  tools  were  selected  for  immediate  evaluation,  the  recommendation  is  for  the  43  tools  to  be  assessed  against 
the  forthcoming  attributes,  either  by  a  user  or  the  developer,  because  of  the  TSE  capability  they  potentially  bring 
through  individual  use  or  in  concert  with  other  tools. 

4.3.  TSE  Tool  Functionality 

After  applying  the  filter  criteria,  the  entire  landscape  of  tools  was  revisited  to  see  if  each  tool  could  be  assigned  a 
primary  function.  The  purpose  for  doing  this  was  twofold:  determine  how  many  decision  analysis  tools  had  been 
carried  over  for  assessment,  and  address  the  observation  that  all  of  the  tools  appeared  to  be  meeting  a  repetitive  set 
of  functions.  The  apparent  functions  are  shown  in  Table  2,  along  with  a  brief  description  and  the  number  of  tools 
that  have  the  function  as  their  primary  and  secondary.  As  expected  based  on  relying  on  the  OR/MS  DAS  Tool 
surveys  for  tool  input.  Decision  Analysis  tools  made  up  a  large  portion  of  the  initial  tool  landscape  for  this 
assessment.  It  should  be  noted  that  although  a  Multi-Disciplinary  Optimization  (MDO)  function  is  specified  in  Table 
2,  ERS  is  not  focused  on  identifying  an  optimum  system  but  rather  on  assessing  resilience  across  multiple  use  cases. 

Table  2.  TSE  tool  functionality 


Function 

Short  description  of  function 

Tools  with 
primary 

Tools  with 
secondary 

Capturing  Value 

Using  weightings,  value  measures,  surveys,  or  other  techniques  to  capture 
user-perceived  value 

3 

9 

Multi-Disciplinary 
Optimization  (MDO) 

Linking  multi-disciplinary  models  for  the  purpose  of  optimizing  a 
constrained  or  unconstrained  objective  function 

12 

5 

Statistical  Data 

Analysis 

Identifying  trends  or  patterns  in  data,  and  developing  models  to  help 
forecast  outputs  based  on  inputs 

25 

15 

Visualization 

Displaying  information  graphically  with  static  or  dynamic  charts  and  tables 

4 

19 

Decision  Analysis 

Evaluating  sets  of  alternatives  against  preferences  on  outcomes 

25 

17 

Project  /  Process 
Portfolio  Management 
and  Simulation 

Assessing  how  much  corporate  value  can  be  achieved  from  project  portfolio 
combinations,  resource  allocation  options,  and  scheduling 

12 

2 

While  the  tools  conveniently  fell  under  these  five  primary  function  categories,  they  should  not  be  considered  to 
perform  only  these  functions.  Tools  can  readily  be  differentiated  from  each  other  when  considering  lower-level 
functions  such  as  process  simulation,  data  mining,  regression,  decision  trees,  brainstorming,  Monte  Carlo  simulation 
(MCS),  sensitivity  analysis,  and  requirements  analysis.  Not  all  tools  were  assessed  to  have  a  secondary  function. 

4.4.  TSE  Tool  Attributes 

Continuing  with  the  reduced  set  of  tools,  the  next  task  was  to  identify  attributes  that  constitute  a  TSE  tool,  and 
that  define  the  functionality  desired  in  such  a  tool.  Referring  back  to  the  sources  in  Table  1,  an  initial  set  of  25 
attributes  was  developed. 

Other  sources  were  then  reviewed  in  the  areas  of  visualization,  optimization,  and  value'^  '^  ’^’'*  '^’^®,  and  the 
industry  surveys^’^  ®  were  more  thoroughly  reviewed  given  their  popularity  and  the  quality  in  their  results  based  on 
the  assumed  standardization  of  their  questions  and  corresponding  attributes. 
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An  interesting  discovery  was  made  when  attempting  to  validate  the  assumed  standardization.  The  OR/MS  Today 
DAS  Surveys  are  set  up  as  online  questionnaires.  A  hyperlink  is  provided  for  tool  vendors  and  developers  to 
describe  their  software  by  entering  in  yes  (“y”))  *10  or  ^  limited  textual  description.  The  2010  Survey^  did  not 
provide  rationale  or  descriptions  of  the  survey  questions.  There  was  also  no  indication  of  the  provenance  of  the 
questions,  other  than  a  short  statement  about  the  industries  from  which  new  questions  were  pulled:  “For  this  year’s 
[2010]  survey...  questions  based  on  the  input  of  decision  analysis  practitioners  from  government,  airline,  oil  and  gas 
and  pharmaceutical  sectors  were  added.”  In  the  2012  Survey^,  the  same  questions  were  provided  to  the  vendors  who 
had  responded  in  previous  years,  as  well  as  to  vendors  familiar  to  the  survey  moderator  —  not  unlike  what  was  done 
in  this  assessment  when  filtering  for  tools  to  include  for  immediate  evaluation.  An  electronic  mail  inquiry  sent  to  the 
2010  and  2012  Survey  moderators  revealed  that  the  origin  of  the  Survey  questions  most  likely  traces  back  to  a  single 
individual,  and  there  is  no  recorded  pedigree  on  the  questions  or  attributes,  nor  are  formal,  objective,  unambiguous 
definitions  available.  Further,  each  moderator  responsible  with  delivering  the  survey  in  a  given  year  is  permitted  to 
update  attributes  as  they  see  necessary  (e.g.,  when  technology  advances  result  in  a  question  or  attribute  being  no 
longer  applicable).  Lastly,  the  tool  vendors  are  permitted  to  “self-assess”  their  product’s  capability  against  the 
attributes,  meaning  the  attributes  are  open  to  interpretation  by  those  completing  the  Survey. 

Based  on  the  above  description  of  the  DAS  Surveys,  the  authors  of  this  paper  have  recommended  to  the  Institute 
for  Operations  Research  and  the  Management  Sciences  (INFORMS)  that  the  2014  OR/MS  Today  DAS  Survey 
moderator  consider  standardizing  the  questions  and  attributes  by  developing  unambiguous  definitions,  possibly 
using  the  work  performed  here  as  a  starting  point. 

After  revisiting  the  initial  TSE  tool  attributes,  the  set  increased  to  91,  practically  eliminating  the  ability  to 
perform  a  value  focused  thinking  approach  at  this  stage.  The  attributes  were  then  grouped  into  common,  higher  level 
affinity  categories,  which  are  shown  in  Table  3  with  a  short  description  and  the  number  of  attributes  per  category.  It 
should  be  noted  that  these  categories  are  a  mix  of  quantitative  and  qualitative,  including  binary  (e.g.,  y  or  n), 
enumerated  (e.g.,  small,  medium,  large),  textual,  currency,  and  integer. 

Table  3.  Top-level  TSE  tool  attribute  categories 


Category 

Short  description  of  category 

Number  of  attributes 

Class 

Is  the  product  better  classified  as  a  tool,  or  a  process 

2 

Usage 

Industries,  market  segments,  or  applications  the  is  tool  used  in 

3 

Operating  Systems 

On  which  operating  systems  can  the  product  operate 

6 

Pricing 

What  is  the  price,  or  pricing  structure,  of  the  product 

4 

Training  Classes  Offered 

What  training  options  available  for  the  product 

4 

T  raining/Experience 
Needed 

Knowledge  level  of  subject  matter  to  effectively  use  the 
product 

3 

Software  Attributes 

Limitations,  expandability,  and  help  options 

8 

Applications 

Decision  making,  preference,  risk,  and  uncertainty  capabilities 

8 

Software  Features 

Meaningful  functions  and  user  experience 

34 

Decision  Algorithms 
Implemented 

How  does  the  product  rank  order  alternatives 

3 

Availability  of  Graphical 
Elicitation  Techniques 

How  does  the  product  capture  values  and  preferences 

9 

Types  of  Output  Display 

How  does  the  product  present  results  for  manipulation 

7 

A  key  capability  worth  noting  is  the  expansion  of  a  tool’s  functionality  through  custom 
coding/programming/scripting,  or  integrating  code  developed  by  others.  The  significance  of  this  capability  can 
readily  be  appreciated  when  understanding  that  a  tool  which  supports  customization  through  programming  can  be 
modified  to  include  any  conceivable  function,  if  the  user  has  the  time  and  knowledge  to  perform  the  programming. 
A  familiar  example  is  the  ubiquitous  Microsoft®  Excel®  spreadsheet  software.  The  built-in  functionality  of  this  tool 
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is  primarily  for  tabulation  and  charting  of  data,  statistical  analysis,  and  numerical  solving.  However,  if  familiar  with 
the  Visual  Basic  for  Applications  (VBA)  language,  the  user  can  expand  and  customize  Excel®  to  perform  any  of  the 
primary  and  lower-level  functions  described  above. 

4.5.  TSE  Tool  Survey 

With  the  tools  and  attributes  identified,  the  next  task  was  to  essentially  self-take  the  survey  for  the  13  tools.  Even 
with  detailed  knowledge  and  experience  on  a  majority  of  these  tools,  gaps  still  existed  in  the  assessment  related  to 
pricing  as  well  as  some  of  the  attributes  pulled  in  from  the  DAS  Surveys.  The  latter  is  driven  by  attribute  ambiguity 
and  subjectivity  as  explained  earlier.  It  is  envisioned  that  the  TSE  Tool  Survey  will  be  fully  populated  by  calling 
upon  experienced  users,  and  possibly  developers,  of  the  identified  tools.  To  do  this,  the  tool  attributes  query  should 
be  formatted  as  a  questionnaire.  The  questions  should  follow  survey  best  practices,  being  as  specific  and  concise  as 
possible  to  remove  ambiguity  and  bias. 

An  important  conclusion  from  this  survey  can  be  drawn  based  on  the  holistic  view  of  TSE  as  described  thus  far: 
having  a  valid  set  of  attributes,  and  an  understanding  of  how  a  cross-section  of  tools  can  satisfy  them,  is  insufficient 
-  what  is  now  needed  is  a  deeper  understanding  of  how  these  tools  are  used  and,  more  importantly,  how  they  can  be 
used  when  performing  TSE  in  support  of  the  Decision  Analysis  Process.  Gaining  this  understanding  will  enable 
users  to  better  assess  if  they  possess  the  appropriate  tools  for  the  TSE  steps  that  they  need  to  perform,  as  well  as 
which  TSE  steps  are  capable  of  being  performed  given  the  tools  in  the  user’s  possession. 

5.  Tradespace  Analysis  Decision  Classification 

This  portion  of  the  effort  involved  looking  across  several  prominent  TSE  research  activities  to  determine  if  there 
exists  a  common,  best  practice  process  for  performing  TSE.  From  these  efforts,  a  consolidated  process  was  outlined 
and  steps  compared  across  the  ERS  DPs  and  two  other  activities  of  interest  to  determine  whether  these  efforts  were 
consistently  performing  a  TSE  process  and,  if  not,  which  steps  were  being  omitted.  A  recommendation  is  offered  for 
mapping  TSE  tools  to  attributes  and  to  process  steps. 

5.1.1.  TSE  Process  Audit 

Thus  far,  tools  were  identified  that  may  be  useful  in  performing  TSE,  their  main  functions  documented,  and  their 
defining  attributes  collected.  However,  these  tools  are  intended  for  use  across  the  system  development  lifecycle,  in 
the  context  of  a  process  for  performing  TSE  and  making  decisions  (Fig.  1).  Therefore,  it  is  important  to  understand 
how  these  tools  can  be  intentionally  used  to  support  a  consistent  TSE  process. 

TSE  research  activities  across  government  and  academia^'’^^’^^’^'^’^^’^®’^’  were  investigated  to  look  for  commonality 
in  process  execution.  Additionally,  TSE  projects^’”"’^^’^^’^”’^'’^^  under  study  within  the  government  were 
investigated  to  better  understand  what  steps  were  being  executed  across  the  full  TSE  activity. 

5.1.2.  TSE  Process  Steps  and  Coverage 

From  the  research  activities  and  pilot  projects,  a  set  of  12  common  steps  emerged,  as  shown  in  Table  4.  It  is 
assumed  that  these  steps  can  be  considered  “TSE  best  common  practice”,  although  this  assumption  is  not  tested 
here.  The  steps  are  not  presented  in  flowchart  format  to  stress  that  a  formal  TSE  process  is  not  being  proposed  but 
rather  existing  processes  consolidated.  The  consolidation  and  subsequent  comparison  to  select  projects  revealed  that: 
1)  TSE  is  commonly  performed  in  steps,  as  assumed;  2)  there  exists  no  formal,  singular,  consistent  process  for 
performing  TSE  across  the  organizations  investigated;  3)  there  exists  no  singular  step  within  TSE  processes  that 
performs  the  action  of  “exploring”  a  tradespace  -  when  such  an  action  is  performed  it  is  done  so  using  prerequisite 
inputs  from  multiple  steps  while  simultaneously  feeding  back,  and  forward,  exploration  results  to  the  decision 
analysis  process;  and  4)  the  ERS  DPs  both  omitted  steps  4  (subject  matter  expert  measures)  and  5  (stakeholder 
value). 
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Table  4.  TSE  best  common  practice  steps 


Step  Description 

1  Determine  mission  scenario(s)  and  their  requirements,  and  keep  them  open  as  long  as  possible 

2  Identify  set  of  operational  performance  characteristics  and  high  level  system  design  variables  that  impact 
operational  requirements 

3  Apply  operational  engagement  models  against  various  mission  scenarios  and  threats  to  identify 
requirements,  MOP,  MOE,  and  other  performance  metrics 

4  Expert  knowledge  teams  determine  values  of  measures  for  given  mission  scenarios  and  requirements 

5  Break  down  stakeholder  values  into  roles,  attributes,  and  specific  tasks 

6  Generate  alternatives  that  meet  requirements  and  constraints,  and  map  stakeholder  values  to  system  design 
variables  using  scalable  multi-physics  based  modeling  design  tools 

7  Create  reduced-order  surrogate  models  to  show  iterative  ability  of  adjusting  scenarios  and  requirements  to 
physical  feasibility 

8  Qualitatively  or  quantitatively  rank  how  alternatives  meet  measures 

9  Perform  a  LCC  estimate  and  lifecycle  schedule  analysis  of  the  system 

1 0  Perform  an  optimization  study  to  determine  the  optimum  feasible  space  that  meets  all  constraints  and  for 

each  course  of  action 

1 1  Determine  courses  of  action  based  on  optimal  feasible  space  and  perform  post-analysis  studies  (operational 
impact  and  gap  analyses) 

12  Perform  case  studies  to  test  for  robustness  and  to  make  sure  that  the  alternative  solutions  are  resilient  in 
changing  operational  environments 


5.1.3.  TSE  Steps,  Tools,  and  Functions 

Consider  that  each  of  the  81  identified  tools,  defined  by  a  subset  of  the  91  identified  attributes,  gives  the  user  an 
ability  to  perform  many  of  the  12  identified  TSE  best  practice  steps.  The  fully  populated  TSE  Tool  Survey  will 
provide  a  link  between  TSE  best  practice  steps,  tools,  and  functions,  allowing  a  user  to  balance  their  TSE  needs  with 
their  organizational  capabilities  and  constraints.  The  survey  will  serve  as  a  database  that  can  be  used  in  different 
ways:  a  user  can  decide  which  tools  they  should  invest  in  by  selecting  the  TSE  steps  that  they  desire  to  perform;  or  a 
user  can  identify  which  TSE  steps  can  be  performed  with  the  tools  currently  available  within  their  organization. 
Knowledge  level,  budget,  and  training  can  serve  as  filters  in  the  user’s  decision  process.  A  recommendation  for 
future  research  is  to  develop  a  decision  analysis  style  user  interface  for  aiding  the  user  in  downselecting  the  most 
appropriate  tools  based  on  their  TSE  needs  and  goals. 

6.  Conclusions  and  Recommendations 

This  effort  used  attributes  of  tradespace  exploration  tools  and  the  steps  in  a  notional  tradespace  exploration 
process  to  illuminate  relationships  between  tool  capabilities  and  process  steps  in  order  to  help  existing  ERS  TSE 
efforts  understand  if  they  are  sufficiently  and  consistently  performing  what  is  considered  to  be  necessary  TSE  for 
ERS.  Although  a  specified,  or  default  standard,  formal  TSE  process  is  not  in  use  for  DoD  programs,  this  effort  has 
provided  insight  into  a  possible  standard  TSE  process  for  ERS  programs.  The  notional  process  requires  further 
investigation  to  determine  if  it  makes  an  improvement  over  current  TSE  processes.  Multiple  tradespace  efforts  in 
government  and  industry  were  reviewed  for  TSE  tools,  TSE  tool  attributes,  and  TSE  process  steps.  A  holistic  view 
of  81  candidate  tradespace  exploration  tools  is  provided.  Tools  were  grouped  into  common  primary  functions. 
Ninety  one  tool  attributes  were  identified.  A  survey  template  was  created  based  on  an  existing  decision  analysis  tool 
survey,  and  the  survey  was  populated  for  several  tools. 

Recommendations  for  future  work  include: 

•  Conduct  a  more  formal  value  focused  thinking  approach,  developing  value  measures  for  attributes  and  functions; 

•  Refine  the  list  of  attributes  and  tools  identified  in  this  effort; 
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•  Complete  the  mapping  of  tools  to  attributes  through  a  survey  of  experienced  users  and  developers.  Request 
rationale  to  support  the  survey  entry.  Possibly  request  an  ordinal  ranking  of  how  well  each  tool  meets  each 
attribute  for  subjective  inputs  to  mimic  the  surveys  referenced  within  this  paper; 

•  Validation  of  the  TSE  steps  identified  in  this  effort; 

•  Adoption  by  INFORMS  of  the  attribute  definitions  developed  here  for  their  next  OR/MS  Today  DAS  survey;  and 

•  Development  of  a  decision  analysis  style  user  interface  to  make  the  database  interactive  for  the  user. 
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